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An  Objective  Method  for  Forecasting  Solar  Flares 


I.  INTRODl  C’TION 

This  report  is  a  continuation  of  an  earlier  study  (Hirman  et  al,  1980)  in  which 
multivariate  discriminant  analysis  (MV’DA)  is  used  in  a  computer  program  to  pro¬ 
duce  an  objective  daily  solar  flare  forecast.  The  essential  feature  of  the  statistics 
package  is  the  comparison  between  a  number  of  input  parameters  and  a  number  of 
output  classes,  in  which  the  discrimination  between  the  classes  in  terms  of  the 
input  parameters  is  maximized  by  constructing  appropriate  classification  functions. 
In  the  application  to  flare  prediction,  the  input  parameters  are  daily  solar  param¬ 
eters  for  each  active  region  on  the  solar  disk,  and  the  output  classes  are  the  levels 
of  flare  activity  occurring  the  following  day  within  the  same  active  regions.  We 
have  used  more  than  two  years  of  data,  of  which  approximately  25  percent  has 
been  used  to  derive  the  classification  functions.  The  latter  are  then  extrapolated 
forward  in  time  to  produce  a  true  forecast. 

The  computer  program,  known  as  BMD07M,  was  originally  written  at  UCLA,  1 
although  the  particular  version  used  here  was  developed  further  by  Seagraves“  to 

Heceived  for  publication  3  Feb  1981 

1.  Dixon,  \V.  J.  (ed.)  (1968)  Biomedical  Computer  Programs,  University  of 

California  Publications  in  Automatic  Computations,  No.  2,  University  of 
California  Press,  p.  214a. 

2.  Seagraves,  P.  H.  (1972)  UBC  BMD07M  Stepwise  Discriminant  Analysis, 

(  niversity  of  British  Columbia  Computing  Centre  1  )ocumentation. 


include  the  Cooley  ami  l.ohnos  classification  procedure,  ’  and  the  I  .achenhrueh 
\-l  technique.  *  The  Cooley  and  l.ohnos  procedure  does  not  assume  uniformm 
of  variance,  and  this  sometimes  results  in  better  classification  scores.  The 
computational  burden,  however,  is  increased  becau.-e  linear  classification 
functions  are  not  possible:  instead,  canonical  variables,  const  meted  from  the 
original  input  parameters,  are  used  as  a  transformation  t  ■  reduce  the  matrix 
dimension  in  the  classification  formulas.  I', he  I .aehenbruch  technique  removes 
bias  when  the  program  classifies  its  own  data  base. 

A  complete  description  of  the  mathematics  is  beyond  the  scope  of  this  report. 
1'he  reader  may  consult  \nderson  and  Kao1’  for  references  on  discriminant 
analysis.  A  discussion  of  the  suitability  of  applying  various  statistical  methods  to 
discrete  input  variables  is  contained  in  Yecchia  et  al.  '  The  latter  point  is  of 
particular  interest  because  the  work  of  Yecchia  et  al  uses  the  same  discrete  data 
base  as  used  herein,  to  produce  solar  flare  probability  forecasts  using  discrimin¬ 
ant  analysis  (without  the  Cooley  and  l.ohnos  procedure!  and  logistic  regression 
analysis. 

An  important  feature  of  the  present  study  is  the  comparison  of  the  objective, 
computer  forecast  with  a  subjective,  conventional  forecast  prepared  during  the 
same  test  period  for  the  same  active  regions  on  the  sun.  Without  such  a  bench¬ 
mark  for  relative  evaluation,  tho  presentation  of  nnv  forecast  method  has  con¬ 
siderably  reduced  merit. 

I)  \T  \ 

The  data  used  herein  were  obtained  from  the  region  analysis  program  at  the 
a'OAA  Space  Environment  Services  Center  (SESC)  in  Boulder,  Colorado.  The 
region  analysis  program  collects  daily  a  variety  of  solar  parameters  for  each 
active  region  on  the  solar  disk.  It  is  important  to  note  that  there  is  no  attempt 


8.  Cooley,  \V.  \V. ,  and  l.ohnes,  P.  R.  (1968)  Multivariate  Procedures  for  the 
Behavioral  Sciences,  Wiley,  New  York. 

1.  Lachenbruch,  P.  A,,  and  Mickey,  M.  R.  (1968)  Technometrics,  10:1. 

а.  Anderson,  T.  U  .  (1958)  An  Introduction  to  .Multivariate  Statistical  Analysis, 

Wiley,  New  Y  ork. 

б.  Rao,  C.  H.  (1974)  Advanced  Statistical  Methods  in  Biometric  Research, 

Ilafner. 

7.  Vecchia,  D.  Caldwell,  G.  A.,  Tryon,  P.  V.,  and  Jones,  R.  H.  (1980) 
in  Sol. -Terres.  Pred.  Proc.,  Vol.  3,  R.  F.  Donnelly  <ed. ),  C-76. 
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in  this  program  to  select  the  more  flare-productive  regions.  The  parameters 
include  radio  and  \-ray  data,  hut  most  are  derived  from  optical  data  supplied 
by  the  I'SAl'/AWS  SOON  system.  The  parameters  contain  information  which  the 
SESC  forecasters  consider  vital  to  the  preparation  of  a  24 -hour  flare  forecast. 
The  present  study  uses  data  for  the  period  1  January  1977  to  21  January  1979, 
containing  (1095  active-region  days  (records)  that  have  been  checked  for  errors 
and  internal  consistency.  Random  scrutiny,  however,  lias  shown  that  errors 
still  remain.  After  several  reassignments  of  parameter  values  and  definitions, 
we  arrived  at  the  form  of  the  data  base  shown  in  Table  1, 


Table  1.  SESC  Region  Analysis  Parameters  (Modified) 


PARAMETER 


ASSIGNED  V  A I  .I  E 


1.  DATE 

2.  REGION  NUMBER 

3.  REGION'S  1TRST  APPEARANCE  LONGITUDE 

4.  CURRENT  LONGITUDE 

5.  N/S  LATITUDE 

6.  CURRENT  LATITUDE 

7.  CARRINGTON  LONGITUDE 

8.  REGION  AGE 

9.  SPOT  CLASS  1 


10.  SPOT  Cl. ASS  2 


r/x .  1 


a .  3 

h .  4 

k . . . r> 


11.  SPOT  CLASS  3 


x .  1 

O .  •> 

i  .  3 

c  . . 4 

13,  MAGNETIC  CLASS 

No  spots . 0 

Alpha .  1 

Beta . . . . .  2 

Beta-Gamma .  3 

Gamma .  4 

Beta-Delta .  5 

Beta-Gamma-Delta .  6 

Gamma-Delta .  7 

13.  MAGNETIC  POLARITY  OF  STRONGEST  FIELD .  (+/-) 

14.  MAGNETIC  FIELD  STRENGTH . . .  (Gauss) 

15.  MAGNETIC  GRADIENTS .  (Gamma/km) 

16.  INTERACTION  WITH  ANOTHER  REGION 

None  . . . . . . . .  0 

Spots  of  opposite  polarity  converge  (from  less 
than  two  degrees  apart) . . .  1 

17.  SUNSPOT  DYNAMICS 

No  spots  or  no  motion .  0 

Coalescing  of  spots  . .  1 


Table  1.  SE.SC  Region  Analysis  Parameters  (Modified)  (continued) 


Spot  rotation . 2 

Relative  motion  between  oppositely  poled  spots . 

18.  STAGE  ()!'  DEVELOPMENT 

No  spots . 0 

Mature  group  (stable)  . . . . .  . .  1 

Decaying . . . .  ......  2 

Growing . 3 

Rapid  decay  (spot  numbers/areas  decrease  by  •>  50  "  )  . .  4 

Rapid  growth  (spot  numbers/ areas  increase  bv  >  301" ) .  f> 

Rapid  growth  (>  100"') . 6 

19.  LEADER/TRAILER  FIELDS 

Structure  not  definite . 0 

Returning  Region  . . 1 

<5  deg  of  neutral  line  and  out  of  phase . .  .  ,  2 

> :i  deg  of  neutral  line  and  in  leader  fields .  8 


>5  deg  of  neutral  line  and  in  trailer  fields 
<.'i  deg  of  neutral  line  and  in-phase . 

20.  RETURNING  REGION 

21.  SECTOR  BOUNDARY  RELATIONSHIP 

22.  ASSOCIATED  FILAMENT 

None . . . 

Filament  unchanged  . . 

Filament  growing . . . . 

Filament  disappeared  within  past  24  hrs  . 
Filament  darkens  or  is  active . 

23.  EMBEDDED  FILAMENT 

None . . . 

Filament  present . 


Active  filament 


Table  1.  SESC'  Hegion  Analysis  Parameters  (Modified)  (continued) 


Table  1.  SESC  Region  Analysis  Parameters  (Modified)  (continued) 


31.  ISOLATED  POLE 

32.  EMERGING  FI.VX 

None,  or  region  is  new . . 

New'  flux  emerges  within  spot  group . 

New  flux  emerges  near  region  (within  5  deg) . 

33.  ARCH  FILAMENT  SYSTEM 

34.  RADIO  BFRST/SWEEP 

None  occurred  . . 

>250  flux  units  at  10  cm . . 

>1000  flux  units  at  10  cm . 

Type  III . 

Type  IV . 

Type  II  and  IV  .  . . . 

F  Burst . . . 

Major/complex  10  cm  burst . 

>1000  flux  units  at  10  cm  plus  a  U  burst,  or 
Type  III  and  II',  or 

250  flux  units  at  10  cm  plus  Type  III  and  IV  . . . 

35.  REGION'S  FIRST  APPEARANCE  (TRANSIT  HISTORY) 

36.  FLARE  HISTORY 

No  flares  have  occurred . 

C  class  flares  have  occurred  . . . 

M  class  flares  have  occurred . 

X  class  flares  have  occurred . 

37.  FLARES  TODAY 


0 

1 


0 

1 

2 

3 

4 

5 

6 
7 


8 


0 

1 

2 

3 


None 


E 


.»  -d  vu*"* 


Most  of  the  parameters  in  1'able  1  have  oeen  assigned  discrete  values 
according  to  categories  which  are  subiectivelv  related  to  increasing  flare 
activity.  This  subjectivity  is  the  weakest  link  in  an\  scheme  utilizing  objective 
procedures  for  producing  a  forecast  solelv  from  data.  In  essence,  the  situa¬ 
tion  merely  allows  the  element  of  subjectivity  to  reside  entirely  in  the  data 
acquisition  process.  Probably,  this  situation  is  preferable  to  having  subiocti- 
vity  introduced  also  in  the  forecast  preparation.  There  are  sev  eral  parameters 
(o.g.  spot  class,  flare  history,  magnetic  class)  for  which  assigned  values  are 
based  upon  quantitative  studies.  Fortunately,  for  perhaps  therefore!  )  these  pararr, 
eters  are  among  those  from  which  the  obiective  forecast  derives  most  of  its  skill. 

Perhaps  the  most  unfortunate  circumstance  is  that  for  a  large  number  of 
records  one  or  more  parameters  is  missing.  In  the  computer  program,  mis¬ 
sing  data  codes  are  replaced  by  averages  for  the  particular  parameter  in  the 
set  of  records  used  in  deriving  the  classification  functions.  Missing  data,  in 
addition  to  errors,  makes  the  testing  of  objective  techniques  difficult,  espec¬ 
ially  for  determining  the  relative  significance  of  various  parameters.  In 
order  to  portray  some  feeling  for  the  degree  of  representation  in  the  data  base 
we  note  the  following:  for  three  commonly  observed  parameters.  Spot  Class 
2,  'Magnetic  Class,  and  "Flares  Today,  "  only  5893  of  the  total  6095  records 
contain  all  three;  if  "Bright  Points,  "  "Spot  Class  3,  "  "Spot  Class  1,  "  "Magnetic 
Gradients,  "  and  "Sunspot  Dynamics"  are  added  to  the  first  three,  only  3732 
records  remain;  and  for  a  total  of  15  of  the  31  usable  parameters,  only  510 


Nevertheless,  we  are  able  to  show  later  that  at  least  some  ol'  these  frequently 
missing  parameters  contain  valuable  predictive  information. 

The  data  base  contains  daily,  region-by-region  entries  for  the  actual  flare 
activity,  in  addition  to  the  official  SESC  subjectively  derived  flare  forecast. 
Thus,  the  information  required  for  objective  forecast  testing,  as  well  as  for 
comparison  with  the  SESC  forecast,  is  contained  in  the  same  base.  Flares  are 
listed  according  to  their  peak  soft  (1-8  ?,)  X-ray  flux  at  1  Al  : 


Class  C: 

io- 

- 

R  <  10“;> 

-a 

watt 

m 

c  lass  M  i 

10'-’ 

-4 

<; 

o 

V 

watt 

m 

_  •> 

C  -lass  X  t 

10 

< 

E 

watt 

m  “  . 

From  the  standpoint  of  geophysical  environment  studies,  the  classes  M  and  X 
are  of  greatest  importance. 

In  addition  to  Table  1,  six  combination  parameters  (Table  2),  derived  from 
certain  original  parameters,  were  included  as  input  parameters.  These  six  were 


Table  2.  Combination  Parameters  (Numbers  in  right-hand 
column  refer  to  original  parameter  number  in  Table  1) 


i 

New  Parameter  No. 

Parameter  Formula 

1 

9*  10-  11 

2 

9.  10*  IP  12 

3 

9-  10.  1 P  32 

4 

14.  15*  (17+25) 

5 

12.  17.  27 

G 

17-  (25+27+28) 

found  to  have  possible  predictive  significance  in  the  earlier  study  where  twenty 

g 

such  combination  parameters  were  tested.  The  derivation  of  combination  param¬ 
eters  is  based  on  intuitions  about  the  form  in  which  predictive  information  might 
be  contained  in  the  data,  and  about  physical  quantities  (e.  g.,  energy  stored  in 
sheared  magnetic  fields)  presumed  relatable  to  flares.  The  subject  of  these  and 
other  combination  parameters  will  be  discussed  in  a  later  section. 


8.  Hirman,  J.  W.,  Neidig,  D.  F,,  Seagraves,  P.  H.,  Flowers,  W.  E.,  and 

Wiborg,  P.  H.  (1980)  in  Sol. -Terrest,  Fred.  Proc.,  Vol.  3,  H.F.  Donnelty 
(ed.),  C-64. 
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The  region  analysis  parameters  for  today  are  independent  of  any  information 
on  flare  activity  occurring  tomorrow;  therefore,  they  can  lie  used  in  practice, 
today,  to  produce  a  Hare  forecast  for  tomorrow,  assuming  that  predictive  informa¬ 
tion  is  present  in  the  parameters.  We  have  used  the  first  \  records  (with  \  =  1500, 
as  described  below)  as  a  training  set'  in  order  to  derive  the  classification  func¬ 
tions  for  three  possible  outcomes;  No  flare,  "  V  flare,  "  and  "M  or  X  flare.  " 

M  and  \  Hares  were  grouped  together  as  a  single  class  in  order  to  reduce  statisti¬ 
cal  noise  caused  by  the  relatively  few  cases  of  larger  Hares.  The  classification 
functions  were  then  applied  to  new  records,  using  only  the  input  parameters,  in 
order  to  produce  a  true  forecast.  The  latter  procedure  was  accomplished  in 
steps  of  250  records  each,  with  the  training  set  sliding  forward  in  time,  2,50 
records  (approximately  one  month)  after  each  step.  Thus,  for  a  1 .500 -record 
training  set,  the  remaining  1  >95-record  test  set  require  If!  individual  subtests 
of  250  records  each  (except  for  the  nineteenth).  This  sliding  base  technique  main¬ 
tains  a  constant  X  records  in  the  training  set,  thereby  assuring  that  the  program 
is  trained  on  recent  data  relative  to  the  test  subset.  This,  combined  with  the 
relatively  small  size  of  the  test  subset,  minimizes  the  effects  of  secular  trends, 
either  of  observational  or  solar  origin,  which  might  he  present  in  the  data. 

The  computer  program  was  trained  on  the  X-ray  class  of  the  largest  event 
(No  flare,  (.'  flare,  or  M  <*  X  flare)  occurring  in  the  region  in  the  24 -hour 
period  following  the  acquisition  date  of  the  input  parameters.  Thus,  the  computer 
forecast  is  expressed  in  terms  of  probabilities  for  the  largest  event  to  be  in  one 
of  those  classes.  The  outcomes  are  mutually  exclusive,  with  the  sum  of  probabili¬ 
ties  over  all  classes  equal  to  unity.  The  XESC  forecast,  however,  is  a  probability 
forecast  for  the  occurrence  of  each  class  of  event;  i.e.,  a  non-exclusive  format. 

In  order  to  assess  the  quality  of  the  computer  forecast,  we  derived  a  comparison 
forecast  in  the  exclusive'  format  by  selecting  the  largest  event  class  in  the 
SESC  forecast  that  was  assigned  a  probability  greater  than  or  equal  to  0.  5.  Al¬ 
though  this  is  not  an  SESC  forecast,  it  is  probably  representative  of  what  would 
be  extant  if  the  SESC  chose  to  cast  their  predictions  in  this  mode. 

In  the  following  test  results  we  present  the  forecasts  according  to  both  the 
standard  multivariate  discriminant  analysis  (MVDA)  and  the  Cooley  and  Lohnes 
procedure  (MVDA/CL).  There  are  important  differences  in  the  character  of 
these  two  forecasts,  which,  as  will  be  shown  later,  may  be  used  to  advantage. 
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1.1  hi'llllim.ii  \  Discussion 

We  are  concerned  mainly  with  the  behavior  of  the  computer  forecast*  rela¬ 
tive  to  the  comparison  forecast  when,  for  example,  changes  are  made  in  the 
size  of  the  training  set,  choice  of  input  parameters,  solar  activity  levels,  and 
percent  of  missing  data.  In  all  cases  we  present  the  computer  forecast  along 
with  the  comparison  forecast  for  the  same  set  of  test  records.  Also  included  is 
a  list  of  input  parameters  submitted  to  analysis,  along  with  their  frequency  of 
selection  in  classifying  the  three  outcomes,  .Note,  however,  that  due  to  the  2.a0- 
reeord  increment  the  training  sets  are  independent  of  each  other  only  when  sepa¬ 
rated  by  six  or  more  subsets. 

\s  a  first  step,  we  eliminated  11  parameters  which  were  not  selected  in  any 
of  thi'  Id  subsets,  following  this,  tfie  program  was  run  again  using  the  remain¬ 
ing  2d  input  parameters.  The  results  are  given  in  Tables  A,  -4,  and  a.  This 
test  ( \)  will  servo  as  an  example  for  the  displays  used  elsewhere  in  tills  report. 

Table  -I  show.-  the  actual  matrix  of  region -day  forecasts  vs  r<  gam-day 
largest  events,  for  the  three  forecasts.  Table  5,  derived  from  the  data  in  Table 
1,  summarizes  the  following: 

T  Percent  of  forecasts  correct  in  the  given  event  class 
['  Percent  of  region-day  largest  events  which  were  forecasted 
\  (P  "  K)/2 

P  Phmatology  (percent  of  the  total  number  of  events  in  the  class) 
l  1  nweighted  mean  of  the  A  s  for  all  three-event  classes 
W  Weighted  mean  forecast  accuracy  (the  sum  of  the  matrix 

diagonal  elements  divided  by  the  total  number  of  forecasts, 
or  events,  in  all  classes) 

Off  1  Percent  of  forecasts  that  are  one  matrix  element  away  from 
the  diagonal 

off  2  Percent  of  forecasts  that  are  two  matrix  elements  away  from 
the  diagonal 

’These  various  scores  are  of  interest  because  of  the  several  ways  in  which 
forecasts  can  he  used.  Kor  example,  the  F  score,  or  percentage  of  forecasts 
that  are  correct,  is  the  quantity  of  interest  to  a  customer  who  cannot  tolerate 
false  alarms.  A  quite  different  requirement  applies,  however,  in  a  situation 
where  surprise  flares  are  unwelcome.  In  the  latter  case,  the  E  score  is  the 
important  measurement.  Of  course,  knowing  the  customer's  need  in  advance 
allows  the  forecast  to  be  biased  either  toward  underprediction,  which  tends  to 
improve  the  T  score,  or  toward  overprediction,  which  improves  the  K  score. 
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Table  3.  Comparison  of  Forecasts --Tost  A 
(1500 -record  training  set) 


l.argest  Event 

I  .argest 

Event  Observed 

Total 

Forecasted 

No  Flare  C 

Ms  X 

Forecasts 

COM  RAMSON 

No  Flare 

3370 

313 

25 

3614 

C 

501 

199 

59 

7  59 

MNX 

77 

83 

63 

2  22 

Total  Events 

3954 

494 

147 

4  595 

MV1)A 

No  Flare 

3349 

190 

26 

3  565 

C 

513 

306 

51 

770 

M  &  X 

93 

98 

70 

260 

Total  Events 

3954 

494 

147 

4  595 

MVDA/CI. 

No  Flare 

3739 

310 

50 

4105 

C 

18  5 

142 

50 

377 

Mfc  X 

30 

36 

47 

113 

Total  Events 

3954 

4  94 

147 

4595 

As  a  measure  of  the  balanced"  accuracy  of  a  forecast  in  a  given  event  class 
we,  therefore,  introduce  the  average  of  K  and  E,  given  by  A. 

The  accuracy  of  a  forecast  is  always  dependent  upon  the  climatology  for 
the  event  being  forecasted.  Higher  climatological  probabilities  tend  to  improve 
the  chances  for  predictions  to  be  correct.  For  example,  it  is  easy  to  predict 
"No  Flare"  with  90  percent  accuracy,  simply  because  no  flare  occurs  in  almost 
90  percent  of  all  active-region  days.  In  comparing  cumulative  scores  between 
forecasts  it  is  imperative  to  note  the  climatology  which  prevailed  during  the 
test  period.  Climatology  is  affected  bv  a  number  of  factors,  including  event 
classification  criteria,  duration  of  forecast  interval,  and  level  of  solar  activity. 
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Table  4.  Parameters  Submitted  to  Analysis  and 
Their  Frequency  of  Selection  in  19  Subsets--Test  A 


Flares  Today 

19 

New  No.  1 

9 

Mag.  Pol. 

5 

Bright  Points 

19 

Mag.  Grad. 

9 

Neut.  I..  Chg. 

5 

New  No.  2 

17 

Mag.  Class 

G 

Spot  Class  3 

4  ! 

Spot  Dvnam. 

12 

Kadio  B/S 

r, 

Spot  Class  2 

3  i 

New  No.  5 

12 

Flare  Hist. 

G 

Spot  Inter. 

3 

Proton  Hist. 

1  1 

New  No.  3 

0 

Emerg.  Flux 

i 

i  ! 

Spot  Class  1 

9 

New  No.  4 

6 

j 

Table  5.  Comparison  of  Forecast  Scores-  - Test  A 


[•‘or  pc  aster 

Event 

F 

E 

A 

C 

I 

\\ 

(  iff  1 

(  'ff  2 

COMPARISON 

No  Flare 

93.4 

85.4 

89.4 

86.  1 

C 

2G.  2 

40.3 

33.  3 

10.  8 

52.8 

79.  2 

18.  6 

2.  2 

MAX 

28.  1 

42.  9 

35.  G 

3.2 

MVP  A 

No  Flare 

93.  9 

34.7 

89.  3 

86.  1 

(’ 

26.  8 

41.7 

34.  3 

10.8 

53.  6 

78.  9 

18.  4 

2.  G 

MX  X 

2G.  9 

47.  G 

37.3 

3.  2 

MY  DA/  Cl, 

No  Flare 

91.  1 

94.6 

92.8 

8G.  1 

C 

37.7 

28.7 

33.2 

10.8 

54.  3 

85.  5 

12.8 

1.7 

MAX 

41.  G 

32.0 

36.  8 

3.  2 

IT 


Ia  essence,  climatology  is  directly  dependent  upon  "bin  size,  "  Failure  to  state 

climatological  conditions  clearly  (an  unfortunately  common  practice)  makes 

9 

intercomparison  of  forecasts  almost  impossible.  It  seems  that  this  point  can¬ 
not  be  emphasized  enough. 

Because  "No  Flare"  constitutes  the  majority  of  situations  on  the  sun,  it  comes 
as  no  surprise  that  solar  flare  forecasts  are  usually  quite  accurate  overall;  i.e. , 
their  weighted  means  (W)  are  high.  It  is  of  greater  interest,  however,  to  predict 
flares  than  quiet  conditions  and,  for  this  reason,  the  unweighted  score  1,  given 
simply  by  the  mean  of  the  A  scores  over  all  classes,  has  been  included  in  Table  5, 

Finally,  we  note  that  if  a  forecast  is  in  error,  it  is  better  to  be  wrong  by  one 
event  class  than  by  two.  Thus,  the  tendency  for  the  off-diagonal  entries  in  the 
matrix  to  cluster  near  the  diagonal  is  an  important  measure  when  comparing  fore¬ 
cast  scores  which  are  similar  otherwise.  Table  5  includes  a  measure  of  this 
error  distribution  in  the  form  of  the  Off  1  and  Off  2  scores. 

The  scores  (F,  E,  and  A)  have  uncertainties  of  approximately  ±1,  *3,  and 
±5  for  No  Flare,  C  Flare,  and  M  &  X  Flare,  respectively.  The  F  and  \V  scores 
have  uncertainties  of  about  ±1.  Thus,  in  terms  of  A  and  F,  the  three  forecasts 
in  Table  5  are  essentially  identical.  The  MVDA/CL  forecast  definitely  excels  in 
the  W  score,  although  this  is  mainly  due  to  its  tendency  for  underprediction,  which 
places  a  large  number  of  forecasts  in  the  No  Flare  column.  The  tendency  for 
underprediction  in  the  MVDA/CL  forecast  is  evident  also  in  the  F  scores  for  C, 
and  M  &  X  flares,  being  significantly  higher  than  the  corresponding  E  scores. 

On  the  other  hand,  both  the  comparison  and  the  MVDA  forecast  are  biased  toward 
overprediction.  Their  overall  similarity  is  quite  striking. 

4.2  Effect  of  Training  Set  Size 

The  number  of  records  to  be  used  in  the  training  set  should  be  large  enough 
to  provide  sufficient  statistics  to  train  the  computer  program,  yet  small  enough 
to  avoid  the  effects  of  trends  in  the  data.  The  optimum  number,  while  not  known 
from  theory,  may  be  determined  empirically  by  varying  the  training  set  size  and 
comparing  the  scores  of  the  resulting  forecasts.  Table  6  shows  the  results  for 
training  sets  of  750  and  2095  records.  Together  with  Table  5  (1500-record  train¬ 
ing  set)  we  find  differences  of  only  small  significance.  A  close  examination  of 


9.  Simon,  P. ,  Smith,  J.  B. ,  Ding,  Y. ,  Flowers,  W. ,  Guo,  Q. ,  Harvey, 

K.  L. ,  Hedeman,  R.,  Martin,  S.  F. ,  McKenna  Lawlor,  S. ,  Lin,  V., 
Neidig,  D. ,  Obridko,  V.  N. ,  Dodson  Prince,  H.,  Rust,  D. ,  Speich,  D. , 
Starr,  A.,  and  Stepanyan,  N.  N.  (1980)  in  Sol. -Terres.  Pred.  Proc,, 
Vol.  2,  R.  F.  Donnelly  (ed.),  p.  287. 
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Table  6.  Comparison  of  Scores  Using  750-Record 
and  2095-Kecord  Training  Sets--Test  1$  and  (' 


Forecaster 

Event 

F 

E 

A 

l' 

VV 

Off  1 

Off  2 

MVDA 

No  Flare 

93.9 

86.0 

89.  9 

750  Records 

C 

28.  4 

41.  3 

34.  8 

53.  3 

80.0 

17,0 

3.0 

M  &  X 

25.  2 

45.  1 

35.  1 

MVDA/CL 

No  Flare 

91.  7 

93.  7 

92.7 

750  Records 

C 

36.  2 

32.7 

34.4 

52.  8 

85.  1 

13.  1 

1.8 

M  &  X 

36.3 

26.0 

31.  2 

MVDA 

No  Flare 

93.  9 

84.0 

89.0 

2095  Records 

C 

25.  9 

42.8 

34.4 

53.  6 

78.  4 

19.4 

2.  2 

M  It  X 

28.  6 

46.4 

37.  5 

MVDA/CL 

No  Flare 

91.0 

95.0 

93.0 

2095  Records 

C 

38.0 

29.4 

33.7 

54.  3 

85.  8 

12.8 

1.4 

M  It  X 

45.8 

26.4 

36.  1 

the  trend  in  the  various  scores,  however,  suggests  that  there  may  be  some 
improvement,  especially  in  the  MVDA/CL  forecast,  as  the  size  of  the  training 
set  is  increased  from  750  to  1500  records.  The  improvement  is  less  certain  in 
increasing  the  set  from  1500  to  2095.  According  to  motivations  which  will  be 
described  later,  the  E  score  is  of  interest  in  the  case  of  the  MVDA  forecast, 
while  the  F  score  is  of  prime  importance  for  MVDA/CL,  Noting  these,  the  U 
scores,  and  the  fact  that  we  do  not  wish  to  make  the  training  set  unnecessarily 
large,  we  have  decided  to  use  1500  records  in  all  training  sets. 

4.3  Inclusion  of  Additional  Combination  Parameters 

Table  4  indicates  that  five  of  the  six  combination  parameters  from  Table  2 
were  retained  for  analysis  after  the  initial  parameter  selection.  Because  several 
of  these  ranked  highly  in  frequency  of  selection  in  Test  A,  we  decided  to  test 
additional  combination  parameters.  As  in  the  case  of  the  original  six,  the  addi¬ 
tional  parameters  were  derived  on  the  basis  of  intuition.  Their  formulas  are 
given  in  Table  7. 

The  20  new  combination  parameters,  in  addition  to  the  20  parameters  used 
in  Test  A,  were  submitted  to  analysis  in  Test  D  (Tables  8  and  9),  It  is  convenient 
to  defer  the  discussion  of  the  latter  to  the  following  section. 


Table  7.  Additional  Combination  Parameters  (Numbers 
in  right-hand  column  refer  to  original  parameter  numbers 
in  Table  1) 


New  Parameter  No. 

Parameter  Formula 

Rates  of  Change 

7 

29  (today)  -  29  (yesterday) 

8 

37  -  37 

9 

(9-  10-  IP  12)  -  (9-  10-  IP  12) 

10 

17  -  17 

11 

(12-  17-  27)  -  (12*  17-  27) 

12 

38  -  38 

13 

9-9 

14 

(9-  10-11)  -  (9-  10-11) 

15 

15-15 

16 

12  -  12 

Parameters  Squared 

17 

292 

18 

372 

19 

(9-  10-  IP  1 2) 2 

20 

172 

21 

92 

22 

(New  7)2 

23 

(New  8)2 

24 

(New  9>2 

25 

(New  10)2 

26 

(New  13)2 

5>n 


Table  8.  Parameters  Submitted  to  Analysis  and  Their  Frequency 
of  Selection  in  19  Subsets--Tests  D,  E,  K,  G,  and  H 


No.  of 

Test 

Parameters 

D 

40 

Flares  Today 

19 

Hadio  B/S 

6 

Flare  Hist. 

2 

New  18 

17 

New  9 

6 

New  4 

2 

New  2 

16 

New  12 

6 

New  23 

2 

Bright  Pts. 

14 

New  1 

5 

Spot  Class  2 

1 

New  19 

12 

Neut.  L.  Chg. 

5 

Emerg.  Flux 

1 

New  15 

10 

New  5 

5 

New  17 

1 

Mag.  Grad. 

9 

New  14 

5 

New  19 

1 

Proton  Hist. 

9 

New  21 

5 

Spot  Class  1 

0 

New  3 

9 

New  22 

5 

New  11 

0 

New  8 

9 

Mag.  Pol. 

4 

New  13 

0 

New  20 

8 

New  7 

4 

New  16 

0 

Mag.  Class 

7 

Spot  Inter. 

3 

New  26 

0 

New  10 

7 

New  25 

3 

Spot  Class  3 

6 

New  20 

2 

a 

20 

See  Table  4 

1 

E 

15 

Flares  Today 

19 

Spot  Class  3 

14 

Hadio  B/S 

6 

Bright  Pts. 

19 

Spot  Dynam. 

13 

Spot  Class  1 

5 

Mag.  Class 

16 

Proton  Hist. 

11 

Mag.  Pol. 

fi 

Mag.  Grad. 

16 

Flare  Hist. 

10 

Emerg.  Flux 

4 

Spot  Class  2 

14 

Neut.  L.  Chg. 

6 

Spot  Inter. 

3 

F 

8 

Flares  Today 

19 

Spot  Class  2 

16 

Spot  Dynam. 

11 

Bright  Pts. 

19 

Spot  Class  3 

14 

Spot  Class  1 

5 

Mag.  Class 

17 

Mag.  Grad. 

13 

G 

5 

Flares  Today 

19 

Mag.  Class 

18 

Spot  Class  3 

15 

Bright  Pts. 

19 

Spot  Class  2 

16 

H 

3 

Flares  Today 

19 

Mag.  Class 

19 

Spot  Class  2 

19 

21 


Table  9.  Effects  of  Reduction  in  the  Number  of  Input  Parameters 


Forecaster 

Number  of 

Parameters 

U 

W 

Off  1 

Off  2 

R 

COMPARISON 

52.8 

79.2 

18.  6 

2,  2 

2.  22 

TEST  D 

MVDA 

40 

53.8 

79.  5 

18.4 

2.  1 

2.  38 

MVDA/ CL 

54.  6 

85.  1 

13.4 

1.  5 

0.73 

TEST  A 

MVDA 

20 

53.6 

78.  9 

18.4 

2.  6 

2.  63 

MVDA/ CL 

54.3 

85.  5 

12.  8 

1.7 

0.60 

TEST  E 

MVDA 

15 

52.4 

78.  1 

18.  7 

3.  2 

2.80 

MVDA/ CL 

53.9 

85.4 

12.8 

1.  8 

0.65 

TEST  F 

MVDA 

8 

52.2 

78.0 

18.6 

3.4 

2.83 

MVDA/ CL 

53.  3 

85.0 

13.4 

1.6 

0.71 

TEST  G 

MVDA 

5 

53.0 

77.6 

18.  5 

3.  9 

3.02 

MVDA/ CL 

53.7 

84.4 

14.0 

1.7 

0.83 

TEST  H 

MVDA 

3 

51.  5 

78.2 

17.  5 

4.3 

2.  72 

MVDA/ CL 

53.  5 

85.2 

13.  3 

1.  5 

0.61 

4.4  Reduction  in  the  IN  umber  of  Parameters 

The  computer  forecast  was  subjected  to  a  series  of  reductions  (Tests 

E,  F, 

G,  and  H)  in  the  number  of  input  parameters,  according  to  Table  8,  with  the 
corresponding  forecast  results  summarized  in  Table  9.  Table  9  displays  the 
effects  of  parameter  reduction  beginning  with  40  parameters  and  ending  with 
only  three.  In  addition  to  the  previously  used  scores  we  introduce  R,  the  ratio 
of  the  number  of  matrix  entries  below  the  diagonal  to  the  number  above  the 
diagonal.  This  ratio  provides  a  measure  of  the  asymmetry  of  the  forecast,  with 
values  greater  than  unity  indicating  overprediction,  and  values  less  than  unity 
indicating  underprediction. 

Table  9  clearly  illustrates  that  the  reduction  in  the  number  of  parameters 
has  a  small  but  unfavorable  effect  on  the  computer  forecasts.  We  may  regard 
the  tendencies  for  R  to  depart  further  from  unity,  for  Off  2  to  increase,  and  for 
U  to  decline,  as  evidence  for  progressively  worsening  forecasts.  These  three 
effects  are  most  noticeable  in  the  MVDA  forecast,  while  the  latter  effect  alone  is 
marginally  evident  in  MVDA/ CL. 


The  effects  of  the  parameter  reduction  are  offset  by  the  increase  in  the 
number  of  records  containing  all  or  most  of  the  parameters  submitted  for  anal¬ 
ysis  in  tiie  reduced  sets.  This  improvement  in  representation  occurs  because 
in  the  reduction  steps  we  usually  eliminated  those  parameters  that  were  least 
significant;  i.o.,  those  chosen  least  often  in  the  subsets  of  the  previous  test; 
and,  generally,  the  lower  the  significance  of  a  parameter,  the  more  often  it  is 
missing  from  the  data  base.  It  is  concluded,  therefore,  that  the  decline  in  fore¬ 
cast  quality  in  Table  u  would  have  been  more  pronounced  had  all  parameters 
been  present  in  all  records.  This  proves  that  there  is  valuable  predictive  infor¬ 
mation  contained  in  at  least  some  of  the*  less  significant  parameters.  It  is 
emphasized  that,  perhaps  to  a  large  degree,  the  lower  significance  of  these 
parameters  is  due  only  to  their  frequent  absence  from  the  data  base. 

A  final  word  must  be  noted  regarding  the  combination  parameters.  Table  8 
indicates  that  a  number  of  these  new  parameters  have  been  selected  by  the  com¬ 
puter  program  as  significant  in  classifying  the  outcomes,  hue  to  the  complex 
intorcorrelations  among  various  parameters,  however,  in  addition  to  possible 
variance  stabilization  effects  and  other  statistical  phenomena,  we  do  not  fully 
understand  the  true  significance  of  these  comnination  parameters,  (questions 
such  as  tins  probably  must  await  further  testing  on  data  bases  containing  fewer 
missing  parameters. 

1  j  IVsts  on  it  hill  I  \  Kt>|ireseiit«*il  Data  Llaso 

The  most  important  test  of  the  computer  forecast  is  achieved  in  the  case 
where  all  the  parameters  submitted  to  analysis  are  present  in  all  records  of  the 
data  base.  Such  a  test,  using  the  full  set  of  parameters,  is  impossible  with  the 
presently  available  data.  A  test  can  do  made  on  a  fully  represented  base,  how¬ 
ever,  if,  for  example,  only  eight  parameters  are  used,  and  we  are  willing  to 
accept,  a  reduced  base  of  37:12  records,  of  which  only  2232  remain  in  the  test  set. 
Such  a  test  (I)  was  performed,  and  the  results  are  shown  in  Tables  10,  11,  and  12. 

Test  I  shows  a  dramatic  improvement  in  the  MVDA/CL  computer  forecast 
in  all  scores,  while  the  .MVUA  and  comparison  forecasts  show  smaller  improve¬ 
ments.  These  improvements  occur  despite  the  somewhat  lower  flare  climatology 
that  applies  to  this  particular  test  set.  The  fact  that  the  comparison  (subjec¬ 
tive)  forecast  scores  are  higher  indicates  that  the  more  complete  observational 
coverage  during  this  sample  of  records  somehow  benefits  the  subjective  methods 
also. 

Due  to  the  reduced  number  of  records,  the  errors  associated  with  the  Test  I 
scores  are  about  50  percent  higher  than  those  stated  earlier.  Nevertheless,  there 
now  seems  no  question  that  the  IVIVDA/CL  forecast  is  superior  to  the  others. 


23 


Table  10.  Comparison  of  Forecasts  l  sing  a  F  ully 
Represented  Data  Base- -Test  1  (1100-rpcord  training  set) 


Largest  Event 

Largest  Event  Observed 

Tot  al 

Forecasted 

No  Flare 

C 

i\  I  &  X 

F  o  recasts 

COMPARISON 

No  Flare 

17.14 

90 

8 

1812 

C 

193 

70 

24 

287 

M  &  X 

28 

31 

34 

93 

Total  Events 

1971 

191 

6G 

2232 

MVDA 

No  Flare 

1707 

G7 

10 

1784 

C 

232 

92 

24 

348 

M  &  X 

3G 

32 

32 

100 

Total  Events 

1975 

19 1 

66 

2232 

MVDA/ CL 

No  Flare 

1829 

97 

14 

1940 

C 

141 

82 

33 

260 

M  &  X 

1 

12 

19 

32 

Total  Events 

197  5 

191 

66 

Table  11.  Parameters  Submitted  to  Analysis  and 
Their  Frequency  of  Selection  in  9  Subsets --Test  I 


Flares  Today 

9 

Mag.  Class 

8 

Spot  Class  2 

5 

Bright  Pts. 

9 

Mag.  Grad. 

8 

Spot  Class  1 

1 

Spot  Class  3 

8 

Spot  Dynam. 

8 
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Fable  lb.  (dm 

pari  son  < 

if  Forecast 

Score- - -  fed  1 

Forecaster 

Event 

K 

K  A 

C  1  V.  <  Iff  1 

<  iff  9 

COMPARISON 

No  Flare 

94.7 

88.8  91.8 

88.  :> 

: 

c: 

24.4 

4  0.  0  40.  .) 

8.0  Y7.  7  89.9  17.1 

1.0 

M  \  X 

u>.  r; 

71.  7  44.  1 

8.  0 

MVDA 

No  Flare 

9 .  7 

Hfi.4  91.1 

88.  7 

( ’ 

Jfi.  4 

48.9  87..; 

8.0  70.  9  89.0  I’..:* 

4.1 

M  •; 

\2.  9 

R;.7  40.9 

o 

M\  1).\/CC 

No  Flan- 

!,4 .  ■; 

•»9.  r,  !)•{. 

88.  ■ 

C 

.  ■;  1 .  •") 

49.9  97.9 

8.  0  78.  9  80.  7  19.9 

0.  7 

M  X 

•  •  ‘ .  1 

98.  8  44.1 

8.0 

:>.  conci.i  sions  \M) km:ommi.m>\tion> 

i'i)f  conclusions  of  this  study  may  be  summarized  as  follows: 

i.  The  standard  MVDA  forecast  is  very  similar  to  the  comparison  forecast 
used  in  this  stum  in  terms  of  overall  accuracy  and  bias  toward  overprediction. 

9.  Tlie  MVl)A/fI..  forecast  is  superior  overall  to  either  the  MYI)A  or  the 
comparison  forecast,  and  is  biased  toward  underprediction. 

The  optimum  sire  for  the  training  set  is  probably  about  1.700  records 
for  the  climatoli  gies  that  prevailed  during  1977  and  1078. 

4.  Flares  l'oday  is  the  most  valuable  prediction  parameter  in  the  data 
base  used  here,  with  the  Bright  Points’  parameter  a  verv  close  second. 

( ither  important  parameters  are  Magnetic  Class,  Magnetic  Gradient,  Spot 
Class,  and  Sunspot.  Dynamics. 

a.  Combination  parameters,  although  their  role  is  not  fully  understood, 
seem  to  improve  forecast  scores. 

0.  Some  of  the  often  missing  parameters  (which  probably,  therefore,  only 
appear  to  be  less  significant  as  predictors)  contain  valuable  predictive  informa¬ 
tion.  Probable  candidates  include  Radio  Burst/ Sweep,  \eutral  Fine  Changes, 
Neutral  Cine  Complexity,  and  Emerging  Flux. 

The  MYDA/CC  procedure  may  be  capable  of  producing  forecasts  superior 
to  any  presently  available  using  conventional,  subjective  techniques.  It  has  been 
shown  that  its  skill  becomes  markedly  evident  when  complete  parameter  repre¬ 
sentation  is  achieved  in  the  data  base.  On  the  basis  of  this,  we  predict  that  with 


improvements  in  data  consistency,  as  well  as  the  inclusion  of  new,  objective 
parameters  in  the  future,  the  computer  forecast  scores  will  continue  to  improve. 


This  study  has  led  us  to  make  the  following  recommendations  concerning 
the  use  of  the  two  computer  forecasts: 

1.  Provide  a  flare  forecast  derived  from  MYDA/CL  for  those  customers 
who  cannot  tolerate  false  flare  alarms  (note  the  comparison  of  F  scores  in 
Table  12). 

2.  Provide  a  flare  forecast  derived  from  standard  MVDA  for  those  cus¬ 
tomers  who  need  to  be  forewarned  of  flares  as  often  as  possible  (compare  E 
scores  in  Table  12). 

3.  Improve  the  coverage  for  the  parameters  in  Table  1  that  are  deemed 
less  significant  by  virtue  of  their  frequent  absence  in  the  data  base. 

4.  Improve  the  objectivity  and  consistency  of  all  parameters. 
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